<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Lexer API</title>
<link rel="stylesheet" href="../../../../../../../doc/src/boostbook.css" type="text/css">
<meta name="generator" content="DocBook XSL Stylesheets V1.79.1">
<link rel="home" href="../../../index.html" title="Spirit 2.59">
<link rel="up" href="../reference.html" title="Reference">
<link rel="prev" href="lex_basics.html" title="Basics">
<link rel="next" href="primitives.html" title="Token definition Primitives">
</head>
<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
<table cellpadding="2" width="100%"><tr>
<td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../../../boost.png"></td>
<td align="center"><a href="../../../../../../../index.html">Home</a></td>
<td align="center"><a href="../../../../../../../libs/libraries.htm">Libraries</a></td>
<td align="center"><a href="http://www.boost.org/users/people.html">People</a></td>
<td align="center"><a href="http://www.boost.org/users/faq.html">FAQ</a></td>
<td align="center"><a href="../../../../../../../more/index.htm">More</a></td>
</tr></table>
<hr>
<div class="spirit-nav">
<a accesskey="p" href="lex_basics.html"><img src="../../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../reference.html"><img src="../../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="primitives.html"><img src="../../../../../../../doc/src/images/next.png" alt="Next"></a>
</div>
<div class="section">
<div class="titlepage"><div><div><h4 class="title">
<a name="spirit.lex.reference.lexer_api"></a><a class="link" href="lexer_api.html" title="Lexer API">Lexer API</a>
</h4></div></div></div>
<h6>
<a name="spirit.lex.reference.lexer_api.h0"></a>
          <span class="phrase"><a name="spirit.lex.reference.lexer_api.description"></a></span><a class="link" href="lexer_api.html#spirit.lex.reference.lexer_api.description">Description</a>
        </h6>
<p>
          The library provides a couple of free functions to make using the lexer
          a snap. These functions have three forms. The first form, <code class="computeroutput"><span class="identifier">tokenize</span></code>, simplifies the usage of a stand
          alone lexer (without parsing). The second form, <code class="computeroutput"><span class="identifier">tokenize_and_parse</span></code>,
          combines a lexer step with parsing on the token level (without a skipper).
          The third, <code class="computeroutput"><span class="identifier">tokenize_and_phrase_parse</span></code>,
          works on the token level as well, but additionally employs a skip parser.
          The latter two versions can take in attributes by reference that will hold
          the parsed values on a successful parse.
        </p>
<h6>
<a name="spirit.lex.reference.lexer_api.h1"></a>
          <span class="phrase"><a name="spirit.lex.reference.lexer_api.header"></a></span><a class="link" href="lexer_api.html#spirit.lex.reference.lexer_api.header">Header</a>
        </h6>
<pre class="programlisting"><span class="comment">// forwards to &lt;boost/spirit/home/lex/tokenize_and_parse.hpp&gt;</span>
<span class="preprocessor">#include</span> <span class="special">&lt;</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">spirit</span><span class="special">/</span><span class="identifier">include</span><span class="special">/</span><span class="identifier">lex_tokenize_and_parse</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">&gt;</span>
</pre>
<p>
          For variadic attributes:
        </p>
<pre class="programlisting"><span class="comment">// forwards to &lt;boost/spirit/home/lex/tokenize_and_parse_attr.hpp&gt;</span>
<span class="preprocessor">#include</span> <span class="special">&lt;</span><span class="identifier">boost</span><span class="special">/</span><span class="identifier">spirit</span><span class="special">/</span><span class="identifier">include</span><span class="special">/</span><span class="identifier">lex_tokenize_and_parse_attr</span><span class="special">.</span><span class="identifier">hpp</span><span class="special">&gt;</span>
</pre>
<p>
          The variadic attributes version of the API allows one or more attributes
          to be passed into the API functions. The functions taking two or more attributes
          are usable when the parser expression is a <a class="link" href="../../qi/reference/operator/sequence.html" title="Sequence Parser (a &gt;&gt; b)">Sequence</a>
          only. In this case each of the attributes passed have to match the corresponding
          part of the sequence.
        </p>
<p>
          Also, see <a class="link" href="../../structure/include.html" title="Include">Include Structure</a>.
        </p>
<h6>
<a name="spirit.lex.reference.lexer_api.h2"></a>
          <span class="phrase"><a name="spirit.lex.reference.lexer_api.namespace"></a></span><a class="link" href="lexer_api.html#spirit.lex.reference.lexer_api.namespace">Namespace</a>
        </h6>
<div class="informaltable"><table class="table">
<colgroup><col></colgroup>
<thead><tr><th>
                  <p>
                    Name
                  </p>
                </th></tr></thead>
<tbody>
<tr><td>
                  <p>
                    <code class="computeroutput"><span class="identifier">boost</span><span class="special">::</span><span class="identifier">spirit</span><span class="special">::</span><span class="identifier">lex</span><span class="special">::</span><span class="identifier">tokenize</span></code>
                  </p>
                </td></tr>
<tr><td>
                  <p>
                    <code class="computeroutput"><span class="identifier">boost</span><span class="special">::</span><span class="identifier">spirit</span><span class="special">::</span><span class="identifier">lex</span><span class="special">::</span><span class="identifier">tokenize_and_parse</span></code>
                  </p>
                </td></tr>
<tr><td>
                  <p>
                    <code class="computeroutput"><span class="identifier">boost</span><span class="special">::</span><span class="identifier">spirit</span><span class="special">::</span><span class="identifier">lex</span><span class="special">::</span><span class="identifier">tokenize_and_phrase_parse</span></code>
                  </p>
                </td></tr>
<tr><td>
                  <p>
                    <code class="computeroutput"><span class="identifier">boost</span><span class="special">::</span><span class="identifier">spirit</span><span class="special">::</span><span class="identifier">qi</span><span class="special">::</span><span class="identifier">skip_flag</span><span class="special">::</span><span class="identifier">postskip</span></code>
                  </p>
                </td></tr>
<tr><td>
                  <p>
                    <code class="computeroutput"><span class="identifier">boost</span><span class="special">::</span><span class="identifier">spirit</span><span class="special">::</span><span class="identifier">qi</span><span class="special">::</span><span class="identifier">skip_flag</span><span class="special">::</span><span class="identifier">dont_postskip</span></code>
                  </p>
                </td></tr>
</tbody>
</table></div>
<h6>
<a name="spirit.lex.reference.lexer_api.h3"></a>
          <span class="phrase"><a name="spirit.lex.reference.lexer_api.synopsis"></a></span><a class="link" href="lexer_api.html#spirit.lex.reference.lexer_api.synopsis">Synopsis</a>
        </h6>
<p>
          The <code class="computeroutput"><span class="identifier">tokenize</span></code> function is
          one of the main lexer API functions. It simplifies using a lexer to tokenize
          a given input sequence. It's main purpose is to use the lexer to tokenize
          all the input.
        </p>
<p>
          Both functions take a pair of iterators spanning the underlying input stream
          to scan, the lexer object (built from the token definitions), and an (optional)
          functor being called for each of the generated tokens. If no function object
          <code class="computeroutput"><span class="identifier">f</span></code> is given, the generated
          tokens will be discarded.
        </p>
<p>
          The functions return <code class="computeroutput"><span class="keyword">true</span></code>
          if the scanning of the input succeeded (the given input sequence has been
          successfully matched by the given token definitions).
        </p>
<p>
          The argument <code class="computeroutput"><span class="identifier">f</span></code> is expected
          to be a function (callable) object taking a single argument of the token
          type and returning a bool, indicating whether the tokenization should be
          canceled. If it returns <code class="computeroutput"><span class="keyword">false</span></code>
          the function <code class="computeroutput"><span class="identifier">tokenize</span></code> will
          return <code class="computeroutput"><span class="keyword">false</span></code> as well.
        </p>
<p>
          The <code class="computeroutput"><span class="identifier">initial_state</span></code> argument
          forces lexing to start with the given lexer state. If this is omitted lexing
          starts in the <code class="computeroutput"><span class="string">"INITIAL"</span></code>
          state.
        </p>
<pre class="programlisting"><span class="keyword">template</span> <span class="special">&lt;</span><span class="keyword">typename</span> <span class="identifier">Iterator</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Lexer</span><span class="special">&gt;</span>
<span class="keyword">inline</span> <span class="keyword">bool</span>
<span class="identifier">tokenize</span><span class="special">(</span>
    <span class="identifier">Iterator</span><span class="special">&amp;</span> <span class="identifier">first</span>
  <span class="special">,</span> <span class="identifier">Iterator</span> <span class="identifier">last</span>
  <span class="special">,</span> <span class="identifier">Lexer</span> <span class="keyword">const</span><span class="special">&amp;</span> <span class="identifier">lex</span>
  <span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Lexer</span><span class="special">::</span><span class="identifier">char_type</span> <span class="keyword">const</span><span class="special">*</span> <span class="identifier">initial_state</span> <span class="special">=</span> <span class="number">0</span><span class="special">);</span>

<span class="keyword">template</span> <span class="special">&lt;</span><span class="keyword">typename</span> <span class="identifier">Iterator</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Lexer</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">F</span><span class="special">&gt;</span>
<span class="keyword">inline</span> <span class="keyword">bool</span>
<span class="identifier">tokenize</span><span class="special">(</span>
    <span class="identifier">Iterator</span><span class="special">&amp;</span> <span class="identifier">first</span>
  <span class="special">,</span> <span class="identifier">Iterator</span> <span class="identifier">last</span>
  <span class="special">,</span> <span class="identifier">Lexer</span> <span class="keyword">const</span><span class="special">&amp;</span> <span class="identifier">lex</span>
  <span class="special">,</span> <span class="identifier">F</span> <span class="identifier">f</span>
  <span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Lexer</span><span class="special">::</span><span class="identifier">char_type</span> <span class="keyword">const</span><span class="special">*</span> <span class="identifier">initial_state</span> <span class="special">=</span> <span class="number">0</span><span class="special">);</span>
</pre>
<p>
          The <code class="computeroutput"><span class="identifier">tokenize_and_parse</span></code>
          function is one of the main lexer API functions. It simplifies using a
          lexer as the underlying token source while parsing a given input sequence.
        </p>
<p>
          The functions take a pair of iterators spanning the underlying input stream
          to parse, the lexer object (built from the token definitions) and a parser
          object (built from the parser grammar definition). Additionally they may
          take the attributes for the parser step.
        </p>
<p>
          The function returns <code class="computeroutput"><span class="keyword">true</span></code>
          if the parsing succeeded (the given input sequence has been successfully
          matched by the given grammar).
        </p>
<pre class="programlisting"><span class="keyword">template</span> <span class="special">&lt;</span><span class="keyword">typename</span> <span class="identifier">Iterator</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Lexer</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">ParserExpr</span><span class="special">&gt;</span>
<span class="keyword">inline</span> <span class="keyword">bool</span>
<span class="identifier">tokenize_and_parse</span><span class="special">(</span>
    <span class="identifier">Iterator</span><span class="special">&amp;</span> <span class="identifier">first</span>
  <span class="special">,</span> <span class="identifier">Iterator</span> <span class="identifier">last</span>
  <span class="special">,</span> <span class="identifier">Lexer</span> <span class="keyword">const</span><span class="special">&amp;</span> <span class="identifier">lex</span>
  <span class="special">,</span> <span class="identifier">ParserExpr</span> <span class="keyword">const</span><span class="special">&amp;</span> <span class="identifier">expr</span><span class="special">)</span>

<span class="keyword">template</span> <span class="special">&lt;</span><span class="keyword">typename</span> <span class="identifier">Iterator</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Lexer</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">ParserExpr</span>
  <span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Attr1</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Attr2</span><span class="special">,</span> <span class="special">...,</span> <span class="keyword">typename</span> <span class="identifier">AttrN</span><span class="special">&gt;</span>
<span class="keyword">inline</span> <span class="keyword">bool</span>
<span class="identifier">tokenize_and_parse</span><span class="special">(</span>
    <span class="identifier">Iterator</span><span class="special">&amp;</span> <span class="identifier">first</span>
  <span class="special">,</span> <span class="identifier">Iterator</span> <span class="identifier">last</span>
  <span class="special">,</span> <span class="identifier">Lexer</span> <span class="keyword">const</span><span class="special">&amp;</span> <span class="identifier">lex</span>
  <span class="special">,</span> <span class="identifier">ParserExpr</span> <span class="keyword">const</span><span class="special">&amp;</span> <span class="identifier">expr</span>
  <span class="special">,</span> <span class="identifier">Attr1</span> <span class="keyword">const</span><span class="special">&amp;</span> <span class="identifier">attr1</span><span class="special">,</span> <span class="identifier">Attr2</span> <span class="keyword">const</span><span class="special">&amp;</span> <span class="identifier">attr2</span><span class="special">,</span> <span class="special">...,</span> <span class="identifier">AttrN</span> <span class="keyword">const</span><span class="special">&amp;</span> <span class="identifier">attrN</span><span class="special">);</span>
</pre>
<p>
          The functions <code class="computeroutput"><span class="identifier">tokenize_and_phrase_parse</span></code>
          take a pair of iterators spanning the underlying input stream to parse,
          the lexer object (built from the token definitions) and a parser object
          (built from the parser grammar definition). The additional skipper parameter
          will be used as the skip parser during the parsing process. Additionally
          they may take the attributes for the parser step.
        </p>
<p>
          The function returns <code class="computeroutput"><span class="keyword">true</span></code>
          if the parsing succeeded (the given input sequence has been successfully
          matched by the given grammar).
        </p>
<pre class="programlisting"><span class="keyword">template</span> <span class="special">&lt;</span><span class="keyword">typename</span> <span class="identifier">Iterator</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Lexer</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">ParserExpr</span>
  <span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Skipper</span><span class="special">&gt;</span>
<span class="keyword">inline</span> <span class="keyword">bool</span>
<span class="identifier">tokenize_and_phrase_parse</span><span class="special">(</span>
    <span class="identifier">Iterator</span><span class="special">&amp;</span> <span class="identifier">first</span>
  <span class="special">,</span> <span class="identifier">Iterator</span> <span class="identifier">last</span>
  <span class="special">,</span> <span class="identifier">Lexer</span> <span class="keyword">const</span><span class="special">&amp;</span> <span class="identifier">lex</span>
  <span class="special">,</span> <span class="identifier">ParserExpr</span> <span class="keyword">const</span><span class="special">&amp;</span> <span class="identifier">expr</span>
  <span class="special">,</span> <span class="identifier">Skipper</span> <span class="keyword">const</span><span class="special">&amp;</span> <span class="identifier">skipper</span>
  <span class="special">,</span> <span class="identifier">BOOST_SCOPED_ENUM</span><span class="special">(</span><span class="identifier">skip_flag</span><span class="special">)</span> <span class="identifier">post_skip</span> <span class="special">=</span> <span class="identifier">skip_flag</span><span class="special">::</span><span class="identifier">postskip</span><span class="special">);</span>

<span class="keyword">template</span> <span class="special">&lt;</span><span class="keyword">typename</span> <span class="identifier">Iterator</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Lexer</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">ParserExpr</span>
  <span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Skipper</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Attribute</span><span class="special">&gt;</span>
<span class="keyword">inline</span> <span class="keyword">bool</span>
<span class="identifier">tokenize_and_phrase_parse</span><span class="special">(</span>
    <span class="identifier">Iterator</span><span class="special">&amp;</span> <span class="identifier">first</span>
  <span class="special">,</span> <span class="identifier">Iterator</span> <span class="identifier">last</span>
  <span class="special">,</span> <span class="identifier">Lexer</span> <span class="keyword">const</span><span class="special">&amp;</span> <span class="identifier">lex</span>
  <span class="special">,</span> <span class="identifier">ParserExpr</span> <span class="keyword">const</span><span class="special">&amp;</span> <span class="identifier">expr</span>
  <span class="special">,</span> <span class="identifier">Skipper</span> <span class="keyword">const</span><span class="special">&amp;</span> <span class="identifier">skipper</span>
  <span class="special">,</span> <span class="identifier">Attribute</span><span class="special">&amp;</span> <span class="identifier">attr</span><span class="special">);</span>

<span class="keyword">template</span> <span class="special">&lt;</span><span class="keyword">typename</span> <span class="identifier">Iterator</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Lexer</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">ParserExpr</span>
  <span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Skipper</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Attribute</span><span class="special">&gt;</span>
<span class="keyword">inline</span> <span class="keyword">bool</span>
<span class="identifier">tokenize_and_phrase_parse</span><span class="special">(</span>
    <span class="identifier">Iterator</span><span class="special">&amp;</span> <span class="identifier">first</span>
  <span class="special">,</span> <span class="identifier">Iterator</span> <span class="identifier">last</span>
  <span class="special">,</span> <span class="identifier">Lexer</span> <span class="keyword">const</span><span class="special">&amp;</span> <span class="identifier">lex</span>
  <span class="special">,</span> <span class="identifier">ParserExpr</span> <span class="keyword">const</span><span class="special">&amp;</span> <span class="identifier">expr</span>
  <span class="special">,</span> <span class="identifier">Skipper</span> <span class="keyword">const</span><span class="special">&amp;</span> <span class="identifier">skipper</span>
  <span class="special">,</span> <span class="identifier">BOOST_SCOPED_ENUM</span><span class="special">(</span><span class="identifier">skip_flag</span><span class="special">)</span> <span class="identifier">post_skip</span><span class="special">,</span> <span class="identifier">Attribute</span><span class="special">&amp;</span> <span class="identifier">attr</span><span class="special">);</span>
</pre>
<p>
          The maximum number of supported arguments is limited by the preprocessor
          constant <code class="computeroutput"><span class="identifier">SPIRIT_ARGUMENTS_LIMIT</span></code>.
          This constant defaults to the value defined by the preprocessor constant
          <code class="computeroutput"><span class="identifier">PHOENIX_LIMIT</span></code> (which in
          turn defaults to <code class="computeroutput"><span class="number">10</span></code>).
        </p>
<div class="note"><table border="0" summary="Note">
<tr>
<td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="../../../images/note.png"></td>
<th align="left">Note</th>
</tr>
<tr><td align="left" valign="top"><p>
            The variadic function with two or more attributes internally combine
            references to all passed attributes into a <code class="computeroutput"><span class="identifier">fusion</span><span class="special">::</span><span class="identifier">vector</span></code>
            and forward this as a combined attribute to the corresponding one attribute
            function.
          </p></td></tr>
</table></div>
<p>
          The <code class="computeroutput"><span class="identifier">tokenize_and_phrase_parse</span></code>
          functions not taking an explicit <code class="computeroutput"><span class="identifier">skip_flag</span></code>
          as one of their arguments invoke the passed skipper after a successful
          match of the parser expression. This can be inhibited by using the other
          versions of that function while passing <code class="computeroutput"><span class="identifier">skip_flag</span><span class="special">::</span><span class="identifier">dont_postskip</span></code>
          to the corresponding argument.
        </p>
<h6>
<a name="spirit.lex.reference.lexer_api.h4"></a>
          <span class="phrase"><a name="spirit.lex.reference.lexer_api.template_parameters"></a></span><a class="link" href="lexer_api.html#spirit.lex.reference.lexer_api.template_parameters">Template parameters</a>
        </h6>
<div class="informaltable"><table class="table">
<colgroup>
<col>
<col>
</colgroup>
<thead><tr>
<th>
                  <p>
                    Parameter
                  </p>
                </th>
<th>
                  <p>
                    Description
                  </p>
                </th>
</tr></thead>
<tbody>
<tr>
<td>
                  <p>
                    <code class="computeroutput"><span class="identifier">Iterator</span></code>
                  </p>
                </td>
<td>
                  <p>
                    <a href="http://en.cppreference.com/w/cpp/named_req/ForwardIterator" target="_top"><code class="computeroutput"><span class="identifier">ForwardIterator</span></code></a> pointing
                    to the underlying input sequence to parse.
                  </p>
                </td>
</tr>
<tr>
<td>
                  <p>
                    <code class="computeroutput"><span class="identifier">Lexer</span></code>
                  </p>
                </td>
<td>
                  <p>
                    A lexer (token definition) object.
                  </p>
                </td>
</tr>
<tr>
<td>
                  <p>
                    <code class="computeroutput"><span class="identifier">F</span></code>
                  </p>
                </td>
<td>
                  <p>
                    A function object called for each generated token.
                  </p>
                </td>
</tr>
<tr>
<td>
                  <p>
                    <code class="computeroutput"><span class="identifier">ParserExpr</span></code>
                  </p>
                </td>
<td>
                  <p>
                    An expression that can be converted to a Qi parser.
                  </p>
                </td>
</tr>
<tr>
<td>
                  <p>
                    <code class="computeroutput"><span class="identifier">Skipper</span></code>
                  </p>
                </td>
<td>
                  <p>
                    Parser used to skip white spaces.
                  </p>
                </td>
</tr>
<tr>
<td>
                  <p>
                    <code class="computeroutput"><span class="identifier">Attr1</span></code>, <code class="computeroutput"><span class="identifier">Attr2</span></code>, ..., <code class="computeroutput"><span class="identifier">AttrN</span></code>
                  </p>
                </td>
<td>
                  <p>
                    One or more attributes.
                  </p>
                </td>
</tr>
</tbody>
</table></div>
</div>
<table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr>
<td align="left"></td>
<td align="right"><div class="copyright-footer">Copyright © 2001-2011 Joel de Guzman, Hartmut Kaiser<p>
        Distributed under the Boost Software License, Version 1.0. (See accompanying
        file LICENSE_1_0.txt or copy at <a href="http://www.boost.org/LICENSE_1_0.txt" target="_top">http://www.boost.org/LICENSE_1_0.txt</a>)
      </p>
</div></td>
</tr></table>
<hr>
<div class="spirit-nav">
<a accesskey="p" href="lex_basics.html"><img src="../../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../reference.html"><img src="../../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="primitives.html"><img src="../../../../../../../doc/src/images/next.png" alt="Next"></a>
</div>
</body>
</html>
